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Unmanned  Air  Vehicles  (UAVs)  promise  to  change  the  way  we  think  about  aircraft  and 
airspace  and  they  are  being  used  in  many  different  environments;  everything  from 
entertainment  to  search  and  rescue  operations.  One  thing  common  amongst  the  different 
UAV  platforms  is  that  all  use  very  difficult  user  interfaces.  These  interfaces  have  been 
designed  by  engineers  and  roboticists  who  do  not  have  training  in  incorporating  the  needs 
of  end  users.  This  research  involves  developing  an  innovative  interface  for  UAVs  that 
improves  situation  awareness  for  the  user  and  demonstrating  its  effectiveness. 

Existing  UAV  interfaces  are  almost  always  targeted  towards  experienced  pilots, 
engineers,  or  someone  who  is  an  expert  in  both  of  those  fields.  The  result  is  interfaces 
which  are  confusing  and  difficult  to  use  for  someone  who  isn’t  an  expert  in  these  areas. 

The  UAV  pilot’s  interface  is  usually  just  a  copy  of  an  aircraft  cockpit  presented  to  the 
user.  For  someone  with  a  great  deal  of  experience  looking  at  the  various  dials  and  gauges 
this  can  be  helpful,  but  the  experience  of  flying  a  UAV  is  not  completely  analogous  to 
flying  a  manned  aircraft,  which  involves  associating  the  feeling  of  flight  to  the  controls. 
When  flying  a  UAV  there  is  no  feedback  in  the  user’s  body  to  indicate  what  is  happening 
with  the  aircraft,  which  makes  it  difficult  to  understand  what  the  aircraft  is  doing. 

In  contrast  to  a  pilot,  an  engineer  who  is  developing  and  troubleshooting  the  UAV  and  its 
interface  needs  access  to  different  elements  of  the  aircraft’s  status,  such  as  the  status  of 
the  internal  computer,  tuning  parameters,  and  servo  positions.  These  elements  are  usually 
displayed  in  a  list  or  a  table.  Once  the  aircraft  can  fly,  this  information  is  no  longer  as 
important  and  its  presence  in  the  display  causes  unnecessary  complexity  for  UAV  pilots. 

Because  UAVs  have  been  relatively  scarce  assets,  requiring  the  users  to  be  experienced 
pilots  or  engineers  has  not  been  a  significant  problem  in  the  past.  However,  an  increasing 
number  of  UAV  operators,  particularly  for  small  UAVs,  are  neither  pilots  nor  engineers. 
Further,  the  lack  of  personnel  with  technical  skills  fundamentally  limits  the  utility  of 
UAVs  unless  their  interfaces  become  more  readily  usable  by  a  wider  segment  of  the 
population.  What  is  needed  is  an  interface  that  allows  the  non-technical  user  to  quickly 
understand  the  status  of  the  aircraft  and  its  surroundings.  Such  an  interface  would  allow 
UAVs  to  be  used  on  a  much  broader  scale. 

When  approaching  a  new  system,  people  construct  a  mental  model  regarding  how  it 
works.  Mental  models  are  often  formed  with  the  aid  of  metaphors  such  as  that  of  the 
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manned  aircraft  cockpit.  For  a  more  universally-understood  metaphor,  we  have  turned  to 
virtual  environments  of  the  type  represented  in  video  games.  Many  of  the  people  who 
have  the  need  to  fly  UAVs  (such  as  military  personnel)  have  experience  with  playing 
video  games.  They  are  skilled  in  navigating  virtual  environments  and  maintaining 
situation  awareness  in  these  environments.  A  user  interface  that  takes  advantage  of  this 
experience  should  facilitate  better  performance  among  non-technical  users  with  less 
training  than  is  needed  for  existing  systems. 

“Augmented  Virtuality”  allows  us  to  create  just  such  an  interface.  We  have  created  a 
virtual  world  using  terrain  information  and  photographs  of  the  area  in  which  the  operator 
wishes  to  fly  the  UAV.  This  world  is  then  augmented  with  information  from  reality, 
which  in  this  case  is  video  data  from  an  on-board  camera.  The  position  and  orientation  of 
an  avatar  for  the  UAV  is  plotted  in  the  virtual  world  exactly  where  it  is  in  the  real  world. 
The  avatar  changes  its  orientation  as  the  aircraft  banks,  climbs,  and  dives  in  the  real 
world.  The  camera  feed  from  the  UAV  is  fed  into  this  virtual  world  and  gives  a  glimpse 
at  what  is  occurring  in  real  time. 


Figure  1:  Screenshot  from  the  new  interface.  Here  you  can  see  how  the  live  video  image  is 
superimposed  on  the  virtual  world. 

Another  aspect  of  tailoring  the  interface  to  those  with  video  game  experience  is  to  use 
familiar  controls.  Microsoft  has  developed  a  popular  and  versatile  human  input  device 
for  use  with  their  XBox®  video  game  console.  We  are  using  the  XBox  controller  to  map 
the  necessary  flight  controls  to  buttons  or  the  joysticks.  This  mapping  allows  the  system 
to  be  controlled  without  requiring  the  use  of  the  keyboard  or  searching  through  menus. 

To  demonstrate  that  this  new  interface  is  successful  in  improving  the  situation  awareness 
of  a  non-technical  user,  we  are  planning  two  experiments.  These  experiments  will 
measure  the  effectiveness  of  the  interface  both  qualitatively  and  quantitatively.  In  the 
first  experiment,  we  will  observe  participants’  performance  to  identify  issues  with  using  a 
commercially-available  interface  versus  the  Augmented  Virtuality  interface  for 
controlling  the  same  UAV.  Besides  qualitative  performance  observations,  we  will  take 
measures  that  have  been  derived  from  a  detailed  definition  of  situation  awareness.  After 
conducting  the  first  experiment  in  simulation,  the  second  experiment  will  be  carried  out 
on  a  smaller  scale  using  real  UAVs  to  verify  the  results. 


